On Multi-Level Modeling as a remedy against the “Language-as-Fixed-Effect Fallacy”

Statistical analyses in psycholinguistics are often done by two separate ANOVAs for repeated measures, over subjects and over items. Multi-level modeling (MLM), by contrast, combines all random factors, subjects and items and any others, into a single full analysis. This property alone makes multi-level modeling superior over conventional single-level ANOVA techniques. This paper explains the basics of MLM, and it compares multi-level and ANOVA analyses using Monte Carlo simulations. This comparison shows that disaggregation in ANOVA analyses yields F ratios that are too high, which leads to incorrectly reporting small effects as “significant”. As a further demonstration of the possible benefits in data modeling, previously published results were re-analyzed with MLM. Other advantages of MLM are: permissiveness against heteroscedasticity, robustness against missing data, and suitability for latin-square and other incomplete designs. Although technically complex, MLM is likely to be a beneficial tool for research in psycholinguistics.